Compression of tables based on occurrence of values

ABSTRACT

Methods and apparatus, including computer program products, for compression of tables based on occurrence of values. In general, a number representing an amount of occurrences of a frequently occurring value in a group of adjacent rows of a column is generated, a vector representing whether the frequently occurring value exists in a row of the column is generated, and the number and the vector are stored to enable searches of the data represented by the number and the vector. The vector may omit a portion representing the group of adjacent rows. The values may be dictionary-based compression values representing business data such as business objects. The compression may be performed in-memory, in parallel, to improve memory utilization, network bandwidth consumption, and processing performance.

BACKGROUND

The present disclosure relates to data processing by digital computer,and more particularly to compression of tables based on occurrence ofvalues.

Search engines may search large amounts of data in database tables, suchas relational tables, to find results. For massive amounts of data, suchas a combination of tables containing millions of records, processing ofthe data may require lots of hardware resources. For example, largeamounts of random access memory space may be needed to store all therecords that are relevant to executing a user request.

SUMMARY

The subject matter disclosed herein provides methods and apparatus,including computer program products, that implement techniques relatedto compression of tables based on occurrence of values.

In one aspect, columns of dictionary-based compression values aregenerated, the columns are sorted, at least one bit vector is generatedfor at least one of the columns, a number representing occurrences of amost-frequently occurring value of at least one of the columns isgenerated, most-frequently occurring values are removed from the bitvector, and, the numbers and the bit vectors are stored to enablein-memory searches of the data represented by each of the numbers andthe bit vectors. The columns of dictionary-based compression values maybe based on a dictionary of possible values for each column of acolumn-based database and the values may represent structured businessdata. The sorting may include sorting the columns such that a firstcolumn ordered first in an ordering of the columns has a most-frequentlyoccurring value of the first column occurring more frequently thanfrequently-occurring values of other columns. Sorting may furtherinclude sorting the first column such that instances of themost-frequently occurring value of the first column are at one end ofthe first column and sorting the other columns such that instances ofmost-frequently occurring values of at least one of the other columnsare towards ends of the respective other columns. Each of the bitvectors may represent most-frequently occurring values of a respectivecolumn, where each bit represents whether a most-frequently occurringvalue exists.

In a related aspect, at least one bit vector corresponding to amost-frequently occurring value of a column of data is generated for atleast one of the columns, at least one number representing occurrencesof the most-frequently occurring value of the column is generated,most-frequently occurring values are removed from the at least one bitvector, and, the number and the at least one bit vector are stored toenable searches of the data represented by the number and the bitvector. Each bit of the vector may represent whether a value isinstantiated at the corresponding place in the column.

In a related aspect, a number representing an amount of occurrences of afrequently occurring value in a group of adjacent rows of a column isgenerated, a vector representing places at which the frequentlyoccurring value is instantiated in a row of the column is generated, andthe number and the vector are stored to enable searches of the datarepresented by the number and the vector. The vector may omit a portionrepresenting the group of adjacent rows.

The subject matter may be implemented as, for example, computer programproducts (e.g., as source code or compiled code tangibly embodied incomputer-readable media), computer-implemented methods, and systems.

Variations may include one or more of the following features.

Values of columns may be values representing structured business data,which may have data dependencies across a same row of a table. Thebusiness data may include business objects, which may be modeled as setsof joined tables.

Actions may be performed in parallel on multiple hardware servers. Forexample, rows may be distributed across a number of servers and eachserver may be responsible for compressing data in its rows.

Frequently occurring values corresponding to a vector, such as amost-frequently occurring value corresponding to a bit vector, may beremoved from a column to generate a reduced column. A reduced column maybe stored in lieu of a column.

Bit vectors may be generated for each of the columns, for all of thecolumns, or for any subset of the set of columns.

Changes to values of a column may be stored in a delta buffer separatefrom the column and the changes may be integrated asynchronously.

The subject matter described herein can be implemented to realize one ormore of the following advantages. Efficient processing of mass amountsof database data, such as relational tables containing mass amounts ofdata, may require a high level of data compression to retain datavolumes in installed memory (e.g., volatile memory) or on disk storagedevices, and for efficient data flows when moving the data (e.g., from ahard disk drive to memory). Compression may have multiple effects in aninformation processing hardware landscape because reduced data volumesmay require less installed main memory or hard disk capacity, andreduced data flows may place lower demands on processor cache, processorarchitectures, and on network bandwidth. All this may have a beneficialeffect on hardware requirements, response times, and overall systemperformance. Data, such as business data, may be compressed and searchedin memory, as significant compression of the data may allow forcost-effective in-memory processing of the data (e.g., a number ofservers or amount of physical memory may be reduced). A delta buffer maybe used for asynchronous updates of the data and the delta buffer mayassist in enabling index updates to be scheduled to run at times whichdo not hamper performance of searches (e.g., a system may be employed inscenarios where asynchronous, infrequent updates are preferred as atrade-off against improved response time for searching). A combinationof compression techniques may be implemented, which may providebeneficial memory cost reduction depending on a scenario. For example,dictionary-based compression might only be performed for a column ifexpected to reduce a memory footprint of that column, while vector-basedcompression may be performed for other columns where the compression isexpected to reduce a memory footprint.

Details of one or more implementations are set forth in the accompanyingdrawings and in the description below. Further features, aspects, andadvantages will become apparent from the description, the drawings, andthe claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram illustrating a table of structured data, adictionary for a column of the table, an attribute table, and a mainindex.

FIG. 1B is a block diagram illustrating a table of structured data, adictionary for a column of the table, an attribute table, and a deltaindex.

FIG. 1C is an example illustrating generation of result sets from mainand delta indexes.

FIGS. 2A-2B are block diagrams illustrating tables of structured data.

FIG. 3 is a table illustrating a cardinality of attributes and keyfigures.

FIGS. 4A-4B are block diagrams illustrating columns being compressedaccording to vector-based compression.

FIG. 5 is a block diagram illustrating a system to compress data andsearch compressed data.

FIG. 6 is a block diagram of a table illustrating a sorting of dataacross multiple columns.

FIGS. 7A-7B are flowcharts illustrating processes of compressing dataand enabling searches of compressed data.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

In general, in FIGS. 1-7, data may be compressed using a combination oftechniques, which may be referred to as dictionary-based compression,bit vector compression (or, vector-based compression), and sorting bitvector compression (or, shortened vector-based compression). The datamay be structured business data, where the data is structured in thesense that data may be attributes or key figures which are organized ina data structure, such as a table, and attributes or key figures mayhave dependencies. For example, in a table of information, a row mayhave dependencies among data in the row such that data in each of thecolumns of the row is associated with other data in other columns of therow. The data may form a sparse distribution in the sense thatparticular values such as null values of data may be instantiatedexceedingly often across thousands or millions of rows either in part ofthe data structure, such as in certain rows in a table, or across theentire data structure. For example, a column of data having twentymillion entries may include nineteen million null entries, where thenineteen million null entries are located in various rows that are notnecessarily adjacent.

FIG. 1A is a block diagram illustrating a table 105 of structured data,a dictionary 110 for a column of the table, an attribute table 115, andan index 120. In general, FIG. 1A illustrates how a column 125 in thetable 105 may be provided with the dictionary 110 that specifies valueidentifiers 130 (ValueIds) for the values in the column 125, theattribute table 115 listing value identifiers 135 for respectivedocument identifiers 140 (DocIds), and the index 120 listing documentidentifiers 145 for respective value identifiers 150.

The dictionary 110 may be used to provide what may be referred to asdictionary-based compression, which may involve using the dictionary 110to reduce an amount of data stored in a table by representing values inthe table with identifiers that may take up less memory. In general, thedictionary 110 is a list, which may be sorted, of values appearing in acolumn and identifiers of the values (i.e., the value identifiers).

As an example, to reduce the memory or disk space occupied by a columnfrom a data table by means of dictionary-based compression, a sortedlist of different values appearing in a column may be generated and thedifferent values may be numbered. The numbers (implemented, for example,as integers rather than strings that may represent the valuesthemselves) may be used as placeholders for the values in the tableswhere the values appeared. The largest numbers needed to represent thevalues may be noted. If the cardinality C of a column is defined to bethe number of different values appearing in it and if the total lengthof the column is N, then in a case where the value of C is much lessthan N, dictionary-based compression may bring benefits, such as reducedmemory consumption in contrast to storing of the values in the table. Asorted list of C values may be referred to as a dictionary and may beused to look up the values of the numbers appearing in the tablewhenever these values need to be determined, for example, to returnreadable results to a user.

For example, the table 105 includes a column 125, which has values suchas INTEL, ABB, and HP. The dictionary 110 includes value identifiers120, which may represent the different values that may be in the column125. For example, the attribute table 115 includes a value identifierfor each row of the column 125. For example, the first row 160, referredto as document identifier 1 (“DocId 1”), includes a value INTEL that isrepresented in the attribute table 115 with the value identifier 4 basedon the dictionary 110 having the value identifier 4 associated with thevalue INTEL. Values of the column 125 of the table 105 may be replacedwith the value identifiers of the attribute table 115, which may reducememory footprint of the data represented by the column 125. The valueidentifiers in the new table, along with the dictionary 110 may be usedto reconstruct to the values of the column 125.

To facilitate value lookup and thus render contents of the column in aform that is more adapted for query execution, the index 120 may begenerated and that index may replace the column 125. The index 120 is atable of lists of rows of the column 125 organized by their valueidentifier. For example, one list 155 indicates that a fourth valueidentifier is associated with rows 1, 4, and 8 of the column 125.

As an example of a memory impact of a table, a number of rows in a tableT, such as the table 105 of FIG. 1A, may be equal to N being 1,000,000;and a number of bytes required to code each row may be 500, where 500bytes is equal to 4,000 bits, which is equal to 4 kilobits. Withoutcompression, the table T may require 1,000,000 times 500 bytes of space,which is equivalent to 500 megabytes of memory space; and, bandwidthrequired to move the table T in one second may be 4 gigabits per second.

As an example of how the data may be organized, a number of differentvalues in column A of table T may be equal to a cardinality C of A being250. In that example, the dictionary for column A may be a list of Cvalues numbered by integers from 0 to 250. The binary representation ofintegers up to 250 may require eight bits (since two to the eighth poweris equal to 256) which is one byte. For an example table T consisting often columns similar to column A, an average uncompressed column entry inany column occupies fifty bytes (500 bytes divided by ten columns isfifty bytes per column). The dictionary for column A may require about100 kilobits (from the fact that 250 entries×(1 one byte valueidentifier plus 50 bytes to represent the corresponding value in adictionary entry) is about 12 kilobytes, which is about 100 kilobits).Thus, with dictionary-based compression, column A may occupy 1,000,000bytes, which is about one megabyte with a total space required by columnA with dictionary-based compression being about 1.01 megabyte (space ofcompressed column plus corresponding dictionary); whereas a nominalspace occupied by column A without compression may be 50 megabytes. Sothe compression factor may be about 50.

FIG. 1B is a block diagram illustrating a table 170 of structured data,a dictionary 172 for a column of the table, an attribute table 174, anda delta index 176. In general, features of the block diagram of FIG. 1Bmay operate similarly to features of FIG. 1A. For example, both of thetables 105, 170 store records that may be compressed withdictionary-based compression using the dictionaries 110, 172 of FIGS. 1Aand 1B, respectively, as described above.

In contrast to the block diagram of FIG. 1A, the block diagram of FIG.1B includes the delta index 176, which may be used to store changes,which may include additions, modifications, or deletions, to data of acolumn. In particular, the delta index 176 includes a result of changesto data in a compressed column. The dictionary values of the delta index176 may be chronologically ordered, as may be a case of a typical deltaindex. The chronological order may reflect an ordering of changes madeto data over time, as indicated by associations of the values in thedelta index 176 with indicated document identifiers. For example, thelist 178 of document identifiers 1, 4, and 8, which are associated withthe value identifier 3 may indicate that document identifier 1 was addedbefore the document identifier 4, which was added before the documentidentifier 8. The chronological ordering of dictionary values of thedelta index 176 may advantageously enable incremental writes withoutrevision of previous entries.

The table 170 of FIG. 1B may reflect changes to, or deltas of, the table105 of FIG. 1A. For example, each of the rows of the table 170corresponding to the delta index 176 may represent a row to add to thetable 105 corresponding to the main index 120. As another example, wherea row in the table 170 has a same record identifier as a row of thetable 105, the row in the table 170 corresponding to the delta index 176may represent a replacement of the row in the table 105. As anotherexample, a row in the table 170 may represent a difference between a rowhaving a same record identifier in the table 105 (e.g., a positive ornegative incremental difference in values in various columns of a samerecord).

Implementations of the delta index 176 may differ. For example, althoughFIG. 1B shows the delta index 176 as including dictionary-basedcompression of delta values, that need not be the case.

FIG. 1C is an example illustrating generation of result sets from mainand delta indexes. For example, the delta index 176 may be used inconjunction with an index of compressed data of a column (e.g., such asthe main index 120 of FIG. 1A), where the delta index 176 may storechanges to data in the index. Both the delta index 176 and an index ofcompressed data of a column may be searched and results from the twosources may be merged to produce a composite result which reflects thechanges made to the compressed data (e.g., as discussed below withreference to a delta buffer).

In the example, total revenue for a company (labeled “IBM”) iscalculated by finding the total revenue from the main index andincrementing it with the total revenue from the delta index. Inparticular, the command 180 “RETURN TOTAL REVENUE FOR IBM SALES” may bebroken into two operations, by a server program, where the twooperations include a first operation 182 for revenue for IBM from a mainindex (rows 5 and 6 of FIG. 1A correspond to a value identifier of 3,for “IBM,” in the main index, having a total value of 11 euros) and asecond operation 184 for revenue for IBM from a delta index (rows 5 and6 of FIG. 1B correspond to a value identifier of 4, for “IBM,” in thedelta index, having a total value of 10 euros). And, results from thosetwo operations may be merged by an operand 186, where merging mayinclude incrementing the results of the main index with the results ofthe delta index. Increments may be positive or negative. Updates toentries in the main index may be handled by first canceling an old rowthen inserting the updated row, where the cancelation can in oneimplementation be represented by a negative increment in the delta indexand the insert by a positive increment.

FIGS. 2A-2B are block diagrams illustrating tables 202, 204 ofstructured data. In general, a first table 202 represents animplementation of a sales table not being compressed according todictionary-based compression and a second table 204 represents andimplementation of the first table 202 being compressed in accordancewith dictionary-based compression.

The first table 202 includes columns representing different types ofdata and rows representing different records based on a combination ofthe different types of data. For example, the columns represent salesnumbers 206, dates 208, location codes 210, product code 212, a numberof products sold 214, packaging attributes 216, a currency unit 218, atotal value in cents 220, and an invoice number 222. The first row 224includes a combination of information being a record, including, a salesnumber S2551, a date of Sep. 4, 2004, a location code of L164, a productcode of P21191, and the like.

The second table 204 represents values of the columns 206-222 of thefirst table as dictionary-based compressed values. The types ofcompressed values include attributes, but not key figures (for reasonsthat may be irrelevant to compression of the values), and attributes arerepresented by identifiers in dictionaries 232. For example, values ofthe sales numbers 206 of the first table 202 are compressed as six digitinteger values (which for up to about 250 thousand values may berepresented by eighteen bit integer identifiers) in the first column 228of the second table 204, where those integer values may represent avalue in a first dictionary 230 but the number sold key figure values214 of the first table 202 are not represented in a dictionary (and arerepresented as floating-point numbers for reasons that need not berelevant to compression of the values). A first sales identifier 0000 inthe first dictionary 230 represents a value S2500 of the first column228 of the second table 204.

Although FIGS. 2A-2B include a certain type of dictionary-basedcompression, such compression may differ. For example, although in thesecond table 204 of FIG. 2 key figures are not compressed according todictionary-based compression, in some implementations a combination ofkey figures and attributes may be compressed according todictionary-based compression, only key figures may be compressed, or acombination of select key figures and attributes may be compressed.

FIG. 3 is a table 300 illustrating cardinalities of attributes and keyfigures. For example, the table 300 may include a listing ofcardinalities of respective columns of the first and second tables 202,204 of FIGS. 2A, 2B. In addition to illustrating cardinalities, thetable 300 includes for each cardinality the number of bits required tocode the respective columns. For example, the number of bits required tocode column M_3 of cardinality 3 is 2, since 2 to the second power is 4,which is the smallest integral power of 2 that is greater than or equalto 3.

The columns of the table 300 include a first column 305 identifying acolumn of another table, a second column 310 indicating cardinalities ofvalues in associated columns, and, a third column 315 indicating numbersof bits required to code associated columns based on associatedcardinalities. For example, a first entry 325 of the table 300 indicatesthat a column of attributes identified as M_1 has a cardinality oftwenty four, which requires five bits to code in binary (since two tothe fifth power is the smallest integral power of two that is greaterthan or equal to twenty four).

The table 300 may be used to reduce a memory impact of a table bygenerating column widths of values based on cardinalities of theirvalues, which may be used in combination with dictionary-basedcompression.

FIGS. 4A-4B are block diagrams illustrating columns being compressedaccording to vector-based compression. The compression may be referredto as bit vector compression. In general, the compression may involvefinding a most frequent value in a column and representing theoccurrence or non-occurrence of the value using a bit vector for thecolumn. For example, a one may represent an occurrence of the value anda zero may represent a non-occurrence of the value. The compression mayfurther involve generating a number of occurrences of a frequentlyoccurring value and removing occurrences of the frequently occurringvalue from the bit vector to generate a smaller or reduced bit vector.For example, in contrast to the series of block diagrams in FIG. 4A, theseries of block diagrams in FIG. 4B further includes reducing a bitvector of sorted values to a number of occurrences of a frequentlyoccurring value and a shortened bit vector representing other values.The values in the columns of FIGS. 4A-4B may be dictionary-basedcompression values.

In the first series of block diagrams of FIG. 4A, a bit vector 406 isgenerated for a column 402 of values, as indicated by the first arrow404. The bit vector 406 is populated with zeros and ones, as indicatedby the second arrow 408, with zeros representing a non-occurrence of afrequently-occurring value 0000 and one representing an occurrence ofthe value. The frequently occurring value that is used to populate a bitvector may be a most-frequently occurring value or just a value thatoccurs more frequently than other values. Determining whether a valueoccurs frequently may be based on a tally of values from a scan of acolumn of data, or, statistical analysis of a value expected to occurmost frequently (e.g., a null value may be expected to be amost-frequently occurring value in a table of exceptions where onlyexceptional values are non-null). The scope of occurrence fordetermining whether a value occurs frequently may be limited to a columnof data (i.e., frequently occurring values may differ on a column bycolumn basis). The column 402 of values may be compressed, as indicatedby the third arrow 410, by removing occurrences of the most-frequentlyoccurring value. For example, the value 0000 is removed from the column402 to generate a compressed column 412. To reconstitute values from thecolumn 402 based on the compressed column 412, the bit vector 406 may beused to indicate positions of values in the compressed column 412 andpositions of a frequently occurring value.

For example, once dictionary-based compression has been performed,further compression can be achieved by deploying bit vectors forrelevant columns as follows. For a given column A containing afrequently repeated value, such as a null value, a most frequent value Fmay be found in column A and coded using a bit vector V for the column.The bit vector V may have N terms, where N is a positive integerrepresenting a number of rows in column A. If V is written as a columnbeside column A, V may include a 1 beside each appearance of value F inA and a 0 beside any other row in A. The bit vector V may be separatedfrom the column vector A and all rows including value F may be deletedfrom A to generate a compressed column vector A*. The column A may bereconstituted from the compressed column vector A* by reinserting valuesF as specified by the bit vector V, and the uncompressed column ofreadable values can be reconstituted from A by using a dictionary asspecified through a dictionary-based compression technique.

As an example of how reduction in memory may be realized, for an exampletable T with N equal to 1,000,000 rows and having a column A, let themost frequent value F in column A appear 990,000 times in A. The other10,000 values in A may be taken from the remainder of the set ofdifferent values listed in the dictionary for A, which may contain atotal of C being 250 different values. In this example, the bit vector Vfor column A may contain 1,000,000 bits, which is about 1 megabit, whichis about 125 kilobytes. A compressed column A* may include 10,000entries, where each is coded with 8-bit (i.e., 1 byte) integers, to givea space occupancy of 10 kilobytes (i.e., 10,000 entries×1 byte). A totalspace required by column A without compression may be 50 MB (asdiscussed with reference to a non-compressed example column A). A totalspace required with vector compression may include space for adictionary, space for the compressed column A*, and space for the bitvector V. Total space required by a vector-based compression version ofcolumn A may be 147 kilobytes (12 kilobytes for the dictionary, 10kilobytes for a compressed column, and 125 kilobytes for a bit vector).A compression factor of about 340 may be realized (i.e., 50,000kilobytes uncompressed/147 kilobytes compressed in accordance with avector-based compression implementation).

In contrast to the first series of block diagrams of FIG. 4A, the secondseries of block diagrams of FIG. 4B involves sorting a column of data toassist in generating an amount representing a number of occurrences of afrequently occurring value. In the second series of block diagrams ofFIG. 4B, a sorted version of a column 414 is generated as shown by afirst arrow 416 to the sorted column 418. Then, a frequently occurringvalue is represented by a bit vector 422 and the sorted column 418 hasthe frequently occurring value replaced, as shown by a second arrow 420to the bit vector 422 and the reduced column 424, which has a frequentlyoccurring value 0000 removed. A number 428 representing an amount ofoccurrences of a frequently occurring value is generated as shown by athird arrow 426. In addition, the bit vector 422 is reduced to remove agrouping of the frequently occurring value to generate the reduced, orshortened, bit vector 430. As some data might not be sorted to the frontor top of bit vector 422, the shortened bit vector 430 may be used todetermine whether a frequently occurring value occurs later in thereduced column 424. For example, dependencies among data across variouscolumns, sorting rules, or a combination of factors might prevent avalue of a column from being sorted to a grouping of a frequentlyoccurring value. To reconstitute a full column, the number 428 ofoccurrences of the value in a grouping and the shortened bit vector 430may be used, in combination with the reduced column 424.

For example, once a table has been compressed by means ofdictionary-based compression and vector-based compression, a furtherlevel of compression may be possible in cases where many columns havemany instances of frequently occurring values (e.g., many null or zerovalues). Rows in the table may be sorted to bring as many of themost-frequently occurring values in columns to a top of their columns aspossible (e.g., as described with reference to table 600 of FIG. 6), andbit vectors may be generated for the columns. The topmost blocks offrequently occurring values may be replaced in the bit vectors bynumbers recording how often frequently occurring values appear in theblocks (e.g., as shown by the number 428 of FIG. 4B). The use of thenumber may allow for bit vectors to be shortened and an increase of anoverall compression ratio.

As a more detailed example, an example table T may have M columns with amost-frequently occurring value in column 1 being F_1, a most-frequentlyoccurring value in column 2 being F_2, and the like to F_M, where anumber of occurrences of a value F_J in a column J may be written as|F_J| and the column J may be any of the columns (i.e., any of thecolumns from 1 to M). The columns may be numbered such that a columnordering is given by the frequency of the most frequent values F, sothat the column with most F values is first and the column with fewest Fvalues is last. Thus, columns 1 to M may be numbered such that|F_1|>|F_2|> . . . >|F_M| (e.g., as may be the numbering of columns 602in table 600 of FIG. 6).

The rows of table T may be sorted by column I such that all values F_1are on top. The sort order may be indifferent to an internal ordering ofthe top |F_1| rows, which may be sorted by column 2 such that all valuesF_2 are on top. The sort order may now be indifferent to an internalordering of the top block of rows with value F_2, such that these rowsare sorted by a column 3 such that all values F_3 are on top. Thesorting of rows may continue until a topmost block of rows with valueF_(M−1) are sorted to put values F_M on top (e.g., continued for all butthe last column M). All the F_1 rows may be on top, with a large numberof the F_2 rows on top, somewhat fewer F_3 rows on top, and so on, withdecreasing completeness (e.g., as depicted in table 600 of FIG. 6). Thismanner of sorting might not be a theoretically optimal sorting formaximizing a final compression ratio, but may be relatively easy toimplement, such that it runs faster than a more complex approach (e.g.,more efficiently utilizes processing resources), and may often be closeto an optimal sorting.

Follow the detailed example, bit vectors V_1 to V_M may be written forthe columns 1 to M, where each bit vector V_J for a column J includes‘1’ values for an occurrence of values F_J and ‘0’ values for any othervalue. The result may be a set of bit vectors V_J that each start with asolid block of values F_J. For each V_J, the solid block of values F_Jand write in their place a number n_J recording how many bits weredeleted in V_J. For sparse tables T (i.e., tables having a frequentoccurrence of a most-frequent value where instances of the value are notnecessarily adjacent), space occupied by a shortened bit vectors V*_Jplus the numbers n_J may be significantly less than space occupied by afull bit vectors V_J.

Shortened vector-based compression may greatly improve efficiency ofaggregation of columns of values. For example, in cases where all thefrequent values F_J are zero, aggregating the values of initial segmentswith a length n_J of the columns may be trivial (e.g., as n_J×zero iszero), and code exploiting this triviality may be much cleaner andfaster than would have been the case without the shortened vector-basedcompression.

As an example of how much compression may be realized, let a table Tagain be as described above, with N being 1,000,000 rows; the tablehaving 10 columns A_1 through A_10; a most frequent value F_1 in columnA_1 appearing 990,000 times; and another 10,000 values in A_1 containinga total of 250 different values, each at 1 byte. Without shortenedvector-based compression, but with vector-based compression column A*_1for a most-frequently occurring value F_1 may occupy 10 kilobytes and abit vector V_1 may occupy 125 kilobytes.

With shortened vector-based compression, a shortened bit vector V*_1 maycontain 10,000 bits and occupy 1.25 kilobytes. The number n_1representing the block of 1 bits for V_1 for the sorted column A_1 maybe in decimal notation 990,000 and require 20 bits in binary notation(i.e., less than 3 bytes). Total space required by A_1 compressed withshortened vector-based compression may include space for a dictionary,space for the short column A*_1, space for the short bit vector V*_1,and space for the number n_1. So the space for A_1 with shortenedvector-based compression may be less than 27 kilobytes (12 kilobytes, 10kilobytes, 1.25 kilobytes, and 3 bytes).

As described above, total space required by a column A_1 withoutcompression may be 50 megabytes. With shortened vector-basedcompression, a compression factor may be greater than 1800 (50,000kilobytes/27 kilobytes, suitably rounded). Compression factors for othercolumns having shortened vector-based compression, such as columns A_2to A_10 may be less, depending on how many of the values F_J for J beingfrom 2 to 10 are sorted to the tops of their columns, but for sparsetables the overall compression factor can still be high enough to makesuch compression well worthwhile, even allowing for the overhead coderequired to reconstruct the columns and read out selected values ondemand.

For both vector-based compression and shortened vector-basedcompression, some overhead resource consumption (including but notrestricted to processing and memory consumption) may be required tocompress and decompress columns, and to facilitate efficient reads ofvalues in a column without decompressing an entire column. Theadditional overhead may take both space (including but not restricted tomain memory space) and time (e.g., measured in terms of percentageutilization of processor core resources) to run, and the penalty mightbe considered in setting thresholds for an employment of shortenedvector-based compression (e.g., in contrast to vector-based compression,dictionary-based compression, or neither). The thresholds may be setheuristically and by testing on various tables. In implementationsinvolving tables containing mass amounts of data, a selection ofcompression techniques may be used (e.g., different techniques fordifferent columns) to minimize memory consumption in lieu of processingconsumption. By minimizing memory consumption, fewer hardware resources,such as a number of blade servers and an amount of installed physicalmemory, may be required. In addition, overall speed of compressing dataand responding to queries may be improved as a minimal memory footprintmay allow for compressing and searching of data in main memory (e.g.,volatile memory, such as random-access memory, having a response timequicker than a secondary memory, such as a hard disk drive used forpersistent storage), and processing overhead for implementingcompression may be acceptably small.

FIG. 5 is a block diagram illustrating a system 500 to compress data andsearch compressed data. The system 500 includes a search enginemanagement tool 502, hosts 504, and storage 506. In general the system500 may be used to search compressed data of the hosts 504 using thesearch engine management tool 502, and the data may be organized andcompressed by the hosts 504. In addition, the data held in memory at thehosts 504 may be persisted at the storage 506, in either compressed oruncompressed form. The search engine management tool 502 may be anintegrated service with the services that perform searching andcompression, implemented redundantly on each of the hosts 504 (forexample in such a manner as to provide mutual monitoring and backup).

The hosts 504 may be organized such that each of the hosts 504 holds aportion of rows of data. For example, a first host 508 may hold a firstmillion rows and a second host 510 may hold a second million rows.Distribution of rows of data across the hosts 504 may be uniform, whichmay improve parallel processing. For example, this may be done toimprove parallel processing to decompress, search and recompress rows ofdata across the hosts 504. As a result of the distribution of data, eachcolumn of a series of columns from one to M, with M being a positivenumber, may be split into parts one through N, with N being a positiveinteger, and one part being allocated to each host where each host isresponsible for the portion allocated to that host (e.g., responsiblefor indexing and compression or for decompression, search andrecompression of an allocated portion).

A logical index for a table of data may be stored at one of the hosts504 and that logical index may be used to determine where data residesacross the hosts 504 and to coordinate processing. For example, thefirst host 508 includes a logical index 518 that may indicate where rowsof data are located across the hosts 504. As an example of coordinatingprocessing, the search engine management tool 502 may query the firsthost 508 for results responsive to a search and the first host 508 mayuse the logical index 518 to coordinate searching of the hosts 504 andmerging results to provide to the search engine management tool 502.

The hosts 504 may be blade servers that share the storage 506. Thestorage 506 may include an index, such as a first index 512,corresponding to each of one or more tables from a database for whichthe data is compressed at the hosts 504. For example, the tables may befact tables and dimension tables for a multidimensional OLAP (OnLineAnalytical Processing) cube. The indexes may contain a logical indexwith metadata for an index structure and a set of compressed columns.For example, the first index 512 includes a logical index 514, which mayinclude metadata for the data of the hosts 504, and a set of compressedcolumns 516.

Each of the hosts 504 may be responsible for compressing rows of datafor which they are responsible. For example, the first host 508 may beresponsible for compressing rows of data in the compressed columns 520.The compression that is performed may be any of the types of compressiondescribed in this document. A compression scheme, such as shortenedvector-based compression, may be stored with each index at a host andcoordinated by a logical index in the case of a split index.

Each of the hosts 504 also includes a delta buffer. For example, thefirst host 508 includes a first delta buffer 522. The delta buffers maystore any changes to a respective index part of a host. For example, thefirst delta buffer 522 may store changes to data stored in the columns522 of data for a first part of a table. The delta buffers may be usedto store changes to data in the hosts 504 instead of requiring updatesto the data in response to each change (e.g., updates may beasynchronous to avoid hampering performance during queries on the data).That the compressed columns need not be updated for individual changesmay allow for compression to improve overall system performance to agreater degree than if changes were written synchronously to the storedcolumns. For example, processing overhead associated with integratingupdates may be reduced if updates are accumulated in the delta buffersand the accumulated updates used to update the main indexes on a lessfrequent basis, since then the overhead resources for compressing theupdated main index are also consumed on a less frequent basis. Forexample, instead of making a thousand small updates to a column indexand having to decompress and recompress the index each time, thethousand updates can be written to the delta buffer and then all writtentogether to the main index, so that only one cycle of decompression andrecompression is required, thus achieving a thousand-fold reduction inoverhead resource consumption. To find search results, the delta buffermay be searched along with the compressed data in a main index andresults from the delta buffer may be merged with results from the mainindex to generate results that incorporate changes. In someimplementations, depending on implementation and configuration details,each of the hosts 504 may also include one or more delta buffers, forexample, a delta buffer for each index.

FIG. 6 is a block diagram of a table 600 illustrating a sorting of dataacross multiple columns 602. Each of the rows 604 includes values thatare dependent across the columns 602. For example, a first row 608includes a combination of values for structure data that represents abusiness object where each of the values in the first row 608 aredependent on other values in the first row 608 such that a sorting ofthe first column 610 sorts data in the other columns such that thecombination of the values of the first row 608 is maintained. Thebusiness objects may be modeled as sets of joined tables. The modelingas a set of joined tables may be a result of business objects beingdefined in such a way that they can be modeled as sets of joined tables,so a search engine may manipulate the objects by searching over rows ofthe tables and calculating specified joins of the business objects.

The data in the table 600 may be dictionary-based compression values.The data in the table 600 may be a result of a sorting the data to groupmost-frequently occurring values in preparation of a vector-basedcompression technique. For example, the value 0 may be a most-frequentlyoccurring value for each of the columns 602 and the rows may have beensorted such that groupings of the value are generated at a top mostportion of the columns 602.

The sorting of the rows 604 in the table 600 may have taken into accounta most-frequently occurring value across the columns 602. For example, asummation row 606 indicates a number of occurrences of a most-frequentlyoccurring value for each of the columns 602. The columns 602 may havebeen ordered such that a most-frequently occurring value of the firstcolumn 610 occurs more frequently than other frequently occurring valuesof each of the columns 602, and, most-frequently occurring values ofother columns are also ordered such that values occurring morefrequently across the column are ordered from A_2 to A_9. Based on asorting of columns, the rows may be sorted to generate as manyfrequently occurring values in groups at one end of a row, and thatsorting may take into account dependencies across columns.

For example, the columns may be ordered horizontally as A_1 to A_9 byhow many 0 values they contain, as shown by the summation row 606. Thefirst column 610, labeled column A_1, may have all of its rows sorted tobring all the 0 values to the top. Then, a second column 612, labeledcolumn A_2, may have rows 1 to 19 sorted to bring the 0 values of thoserows to the top. The sorting of the second column 612 being restrictedto rows 1 to 19 may be based on maintaining the sorting order of the topmost values of the first column 610 and dependencies of data across arow of data. For example, as rows 1 to 19 of the first column 610includes the most-frequently occurring value of that column, to maintainthe grouping of the most-frequently occurring value in rows 1 to 19,only those rows may be sorted in the second column 612. This techniqueof sorting may be followed for each of the remaining columns. Forexample, the third column 614, labeled column A_3, may have rows 1 to 15sorted to bring 0 values of those rows to the top, where 0 values ofother rows may maintain their position (e.g., row 17 includes a 0value). As other examples, a fourth column 616, labeled column A_4, mayhave rows 1 to 14 sorted to bring 0 values to the top of those rows; afifth column 618, labeled column A_5, may have rows 1 to 10 sorted tobring 0 values of those rows to the top; a sixth column 620, labeledcolumn A_6, may have rows 1 to 8 sorted to bring 0 values of those rowsto the top; a seventh column 622, labeled column A_7, may have rows 1 to7 sorted to bring 0 values of those rows to the top; an eight column624, labeled column A_8, may have rows 1 to 6 sorted to bring 0 valuesof those rows to the top; and, a ninth column 626, labeled column A_9,may have rows 1 to 4 sorted to bring 0 values of those rows to the top.

Each of the sorted columns may be compressed using vector-basedcompression, such as the shortened vector based compression describedwith reference to FIG. 4B. The sorting of columns may optimize memorysavings of compression by pushing larger blocks of frequently occurringvalues to one end of a column, such that, for example, shorter bitvectors may be generated with the sorting than may be generated withoutsuch sorting.

Although the table 600 includes a certain organization of data that maybe a result of a sorting, sorting may differ and the data may differ.For example, although a same value, 0, is a most-frequently occurringvalue for each of the columns 602, tables may differ and differentvalues may be a most-frequently occurring value for each of the columns602. As another example, a most-frequently occurring value for onecolumn may be used for sorting all of the columns, or all of the columnsneed not be sorted.

FIGS. 7A-7B are flowcharts illustrating processes 700, 702 ofcompressing data and enabling searches of compressed data. The processes700, 702 may be implemented by the hosts 504 of FIG. 5. For example eachof the hosts 504 may perform the process 700 on a portion of data forwhich a host is responsible. The data that is compressed may bestructured business data. The data may be compressed and searched inmemory, as significant compression of the data may allow forcost-effective in-memory processing of the data (e.g., a number ofservers or an amount of physical memory may be reduced).

In general, in the process 700 of FIG. 7A, dictionary-based values ofcolumns of data in memory are sorted (704), vectors representingfrequently occurring values of the columns are generated (706), numbersrepresenting frequently occurring values are generated (708), and thenumbers and shortened vectors are stored (710).

Sorting of dictionary-based values (704) may involve sorting from alowest value to a higher value for one or more columns of values. Ifvalues of multiple columns are sorted, the sorting may involve sortingsome columns before others based on an ordering of columns (e.g., anordering based on a number of most-frequently occurring values incolumns of a table). For example, the sorting of columns 602 based onthe order of the columns 602 described with reference to the table 600of FIG. 6 may be performed. The sorting may take into accountdependencies of data across columns. For example, the dictionary-basedvalues may represent data that is structured with dependencies acrosscolumns for a same row and an association of values in a row may bemaintained. The sorting may take place in a server, such as one of thehosts 504 of FIG. 5.

For example, sorting dictionary-based values may involve, for eachcolumn of a table, ordering columns such that a column including amost-frequently occurring value (MFOV) of any column is ordered firstand sorting of other columns is based on records within the topmostrange of a previous column (714).

Vectors representing frequently occurring values of the columns aregenerated (706). The vectors may be bit vectors that represent theoccurrence or non-occurrence of a frequently occurring value in a rowwith a bit. The vectors may be generated for all of the columns or foronly some of the columns. The vectors may be generated for parts of acolumn by each server responsible for a portion of data (e.g.,respective hosts of the hosts 504 of FIG. 5). A frequently occurringvalue might be, but need not be, a most-frequently occurring value, suchas a most-frequently occurring value within a scope of a column. Forexample, a bit vector representation for a most-frequently occurringvalue of each column may be generated (716).

Numbers representing frequently occurring values are generated (708).Each of the numbers may represent a number of occurrences of afrequently occurring value in a column. For example, FIG. 4B includes anumber six (428) indicating six occurrences of a frequently occurringvalue 0000. The number of occurrences may be limited to a number ofoccurrences in a group of the frequently occurring value at one end of acolumn (e.g., at a top or bottom). For example, one column may have agroup of a frequently occurring value at a top, the number may representthe number of occurrences of the value in that group, and the column mayinclude other instances of the value. For example, a number representinginstances of a most-frequently occurring value at a topmost section ofeach column may be generated (718).

Numbers and shortened vectors are stored (710). For example, the numberrepresenting occurrences of a frequently occurring value may be storedwith a shortened vector. For example, a set of a number and a bit vectormay be stored for each column of a table. The shortened vector may be avector representing occurrences of the frequently occurring value withthe occurrences of the value represented by the number removed from thevector to generate the shortened vector. In addition to storing theshortened vector, instances of the frequently occurring value in thegroup of the value may be removed from a column to generate a shortenedor reduced column. For example, top ends of bit vectors includingmost-frequently occurring values may be removed (720), and numbersrepresenting the number of occurrences of most-frequently occurringvalues and shortened bit vectors for each column may be stored (722).Also, the top end of a column including a group of most-frequentlyoccurring values may be removed or all instances (e.g., topmost andother instances, if any) of a most-frequently occurring value may beremoved to generate a shortened column (and, e.g., may be reconstitutedusing the bit vector).

In general, in addition to implementations of sub processes of process700 of FIG. 7A, the process 702 of FIG. 7B further includes generatingdictionary-based compression values (712; e.g., as described withreference to FIG. 1A), shortening vectors representing values of columns(720; e.g., as described in the above paragraph), storing a compressedcolumn in memory (722), and decompressing and recompressing a column asrequired to execute queries (724).

Performing a search on a compressed column may involve loading data of acompressed column into memory (e.g., non-volatile memory from compresseddata in a persistent storage), decompressing the data to a temporarystructure, and selecting rows as specified by the search.

Although the processes 700, 702 of FIGS. 7A and 7B include certainsub-processes in a certain order, additional, fewer, or differentsub-processes may exist and those sub-processes may be in a differentorder. For example, a column may be sorted based on bit vectorrepresentations of most-frequently occurring values of a column, ratherthan sorting an entire column (e.g., only values most-frequentlyoccurring may be sorted to one end of a column and other values need nothave a sorting among them).

As another example, dictionary-based compression, general vector-basedcompression, and shortened vector-based compression may be applied basedon determinations as to whether a type of compression, if any, isexpected to optimize performance (e.g., reduce memory consumption). Forexample, dictionary-based compression might be performed ifdictionary-based values and a dictionary are expected to consume lessmemory than non-dictionary-based values of a column. As another example,only rows having attributes but not key figures may be compressed usingany type of compression.

Although each of the figures describes a certain combination offeatures, implementations may vary. For example, additional, different,or fewer components may be included in the system 500 of FIG. 5. Asanother example, in some implementations, a dictionary need not be usedfor a single column. For example, multiple columns may share adictionary.

The subject matter described herein can be implemented in digitalelectronic circuitry, or in computer software, firmware, or hardware,including the structural means disclosed in this specification andstructural equivalents thereof, or in combinations of them. The subjectmatter described herein can be implemented as one or more computerprogram products, i.e., one or more computer programs tangibly embodiedin an information carrier, e.g., in a machine-readable storage device orin a propagated signal, for execution by, or to control the operationof, data processing apparatus, e.g., a programmable processor, acomputer, or multiple computers. A computer program (also known as aprogram, software, software application, or code) can be written in anyform of programming language, including compiled or interpretedlanguages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program does notnecessarily correspond to a file. A program can be stored in a portionof a file that holds other programs or data, in a single file dedicatedto the program in question, or in multiple coordinated files (e.g.,files that store one or more modules, sub-programs, or portions ofcode). A computer program can be deployed to be executed on one computeror on multiple computers at one site or distributed across multiplesites and interconnected by a communication network.

The processes and logic flows described in this specification, includingthe method steps of the subject matter described herein, can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions of the subject matter describedherein by operating on input data and generating output. The processesand logic flows can also be performed by, and apparatus of the subjectmatter described herein can be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for executing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto-optical disks, or optical disks. Media suitable forembodying computer program instructions and data include all forms ofvolatile (e.g., random access memory) or non-volatile memory, includingby way of example semiconductor memory devices, e.g., EPROM, EEPROM, andflash memory devices; magnetic disks, e.g., internal hard disks orremovable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.The processor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, the subject matter describedherein can be implemented on a computer having a display device, e.g., aCRT (cathode ray tube) or LCD (liquid crystal display) monitor, fordisplaying information to the user and a keyboard and a pointing device,e.g., a mouse or a trackball, by which the user can provide input to thecomputer. Other kinds of devices can be used to provide for interactionwith a user as well; for example, feedback provided to the user can beany form of sensory feedback, e.g., visual feedback, auditory feedback,or tactile feedback; and input from the user can be received in anyform, including acoustic, speech, or tactile input.

The subject matter described herein can be implemented in a computingsystem that includes a back-end component (e.g., a data server), amiddleware component (e.g., an application server), or a front-endcomponent (e.g., a client computer having a graphical user interface ora web browser through which a user can interact with an implementationof the subject matter described herein), or any combination of suchback-end, middleware, and front-end components. The components of thesystem can be interconnected by any form or medium of digital datacommunication, e.g., a communication network. Examples of communicationnetworks include a local area network (“LAN”) and a wide area network(“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other in a logical sense andtypically interact through a communication network. The relationship ofclient and server arises by virtue of computer programs running on therespective computers and having a client-server relationship to eachother.

The subject matter described herein has been described in terms ofparticular embodiments, but other embodiments can be implemented and arewithin the scope of the following claims. For example, operations candiffer and still achieve desirable results. In certain implementations,multitasking and parallel processing may be preferable. Otherembodiments are within the scope of the following claims

1. A computer program product, tangibly embodied in a computer-readablemedium, the computer program product being operable to cause dataprocessing apparatus to perform operations comprising: generatingcolumns of dictionary-based compression values, the columns ofdictionary-based compression values being based on a dictionary ofpossible values for each column of a column-based database and beingstructured business data; sorting the columns such that a first columnordered first in an ordering of the columns has a most-frequentlyoccurring value of the first column occurring more frequently thanfrequently-occurring values of other columns; sorting the first columnsuch that instances of the most-frequently occurring value of the firstcolumn are at one end of the first column; sorting at least one of theother columns such that instances of most-frequently occurring values ofthe at least one of the other columns are towards ends of the respectiveother columns; generating a bit vector for at least one of the columns,each of the bit vectors representing most-frequently occurring values ofa respective column, the generating comprising having each bit of a bitvector represent whether a most-frequently occurring value exists if themost-frequently occurring value exists in a respective row of arespective column; generating a number for each of the columns having anassociated bit vector, the number representing an amount of occurrencesof a most-frequently occurring value of one end of a column; removingfrom each of the bit vectors a representation of the most-frequentlyoccurring value of one end of a respective column based on the numberassociated with a bit vector; and storing the one or more numbers foreach of the one or more bit vectors to enable non-volatile memorysearches of the data represented by each of the numbers and the bitvectors.
 2. The product of claim 1, wherein the dictionary-basedcompression values are values representing structured business datahaving data dependencies across a same row of a table.
 3. The product ofclaim 2, wherein the business data comprises business objects modeled assets of joined tables.
 4. The product of claim 1, wherein the operationsof the product are performed in parallel on a plurality of hardwareservers.
 5. The product of claim 4, wherein the operations furthercomprise removing the most-frequently occurring values from columnscorresponding to the one or more numbers to generate reduced columns andstoring the reduced columns in lieu of the columns.
 6. The product ofclaim 1, wherein the bit vector is generated for each or all of thecolumns.
 7. The product of claim 1, wherein changes to thedictionary-based compression values of the columns are stored in a deltabuffer separate from the columns and the changes are integratedasynchronously.
 8. A computer program product, tangibly embodied in acomputer-readable medium, the computer program product being operable tocause data processing apparatus to perform operations comprising:generating a bit vector corresponding to a value of a column of data,the column of data comprising dictionary-compression based values, thegenerating comprising having each bit of the vector represent that thevalue exists if the value exists in a row of the column and having eachbit of the vector represent the value does not exist if another valueexists in a row of the column; generating a number representing anamount of occurrences of a most-frequently occurring value of thecolumn; removing from the bit vector a representation of themost-frequently occurring value in the column; and storing the numberand the bit vector to enable searches of the data represented by thenumber and the bit vector.
 9. The product of claim 8, wherein thedictionary-based compression values are values representing structuredbusiness data having data dependencies across a same row of a table. 10.The product of claim 8, wherein the operations of the product areperformed in parallel on a plurality of hardware servers in non-volatilememory.
 11. The product of claim 8, wherein the bit vector correspondsto the most-frequently occurring value of the column.
 12. The product ofclaim 8, wherein the operations further comprise removing themost-frequently occurring value from the column to generate a reducedcolumn and storing the reduced column in lieu of the column.
 13. Theproduct of claim 8, wherein changes to the dictionary-based compressionvalues of the column are stored in a delta buffer separate from thecolumn and the changes are integrated asynchronously.
 14. The product ofclaim 8, wherein the operations further comprise: generating thedictionary-based compression values; and sorting the dictionary-basedcompression values of the column.
 15. A method comprising: generating anumber representing an amount of occurrences of a frequently occurringvalue in a group of adjacent rows of a column; generating a vector forthe frequently occurring value of the column, the vector representingwhether the frequently occurring value exists in a row of the column andthe vector omitting a portion representing the group of adjacent rows;and storing the number and the vector to enable searches of the datarepresented by the number and the vector.
 16. The method of claim 15,wherein values of the column represent structured business data havingdata dependencies across a same row of a table.
 17. The method of claim15, wherein the vector corresponds to the most-frequently occurringvalue of the column.
 18. The method of claim 15 further comprisingremoving the frequently occurring value from the column to generate areduced column and storing the reduced column in lieu of the column. 19.The method of claim 15, wherein changes to values of the column arestored in a delta buffer separate from the column and the changes areintegrated asynchronously.
 20. The method of claim 15, wherein thevector is a bit vector.